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ABSTRACT 

In  a  {»-evious  paper,  I  described  a  technique  fw  designing  an  observer  test  in  an  iterative 
manner.  In  field  tests  to  compare  the  observability  of  combat  vehicles,  the  test  designer  must 
select  the  optimum  number  of  observation  opportunities  in  order  to  balance  collecting  enough  ‘lata 
to  draw  valid  conclusitms  against  the  high  cost  of  supporting  vehicles  and  po-sonnel  at  a  test  site. 
The  test  designer  can  select  the  number  of  obsCTvations,  N,  so  that  a  given  experimental  differraice 
in  detectability  will  be  statistically  significant  at  a  given  confidence  level.  Alternatively,  the  test 
designer  can  select  N  so  that  die  {wobability  of  rejecting  a  given  undo-lying  difference  in 
detectability  is  less  than  a  given  amount.  The  test  designer,  however,  goierally  lacks  key 
parameters  for  the  efficient  design  of  the  test.  Namely,  the  designer  lacks  the  detectim 
probabilities  of  the  vehicles  at  each  range.  The  standmd  deviation  of  the  difference  in  detection 
probability  depends  upon  flte  detection  probability  itself  Therefore,  die  test  designer  must  select 
the  numbCT  of  observations  for  eadh  range  based  upon  die  conservative  assumption  diat  the 
probabilities  are  near  50%,  the  probability  for  die  maximum  standard  deviation.  In  the  previous 
paper,  an  iterative  technique  of  test  design  was  described.  In  this  technique,  the  test  designer 
m^ifies  the  test  matrix  as  the  test  progresses.  Early  test  results  yield  estimates  of  the  probability 
of  detection  for  each  vehicle  at  eadi  range.  Based  on  these  estimates,  die  test  designs-  reallocates 
the  number  of  observations  among  the  ranges,  im|H-oving  die  efficiency  of  the  test. 

In  this  paper,  I  present  an  algorithm  to  implemsit  this  iterative  technique.  Initially, 
observation  (^portunities  are  equally  distributed  among  the  ranges  of  vdiicle  sites.  But,  as  the  test 
progresses,  the  observation  opportunities  are  redistributed  among  the  ranges  according  to 
estimates  of  detection  probability.  For  example,  the  fiaction  of  opportunities  at  the  range  near  50% 
probability  is  ina-eased  while  the  fi-action  near  10%  js-obability  is  decreased.  The  algorithm 
handles  the  special  cases  that  occur  i^en  the  estimated  probability  is  1  or  0. 


1.  THE  EXPERIMENTAL  SITUATION 

Figure  1  illustrates  a  typical  test  setup  for  a  test  of  observability.  Observers  are  stationed  at  a 
fixed  site  and  attempt  to  detect  a  vehicle  in  their  field  of  view.  For  each  observation  opportunity, 
the  test  personnel  record  the  number  of  detections.  In  analyzing  the  data,  the  analyst  groups  the 
observations  into  range  bins  and  con^ares  the  proportion  of  detections  for  each  test  vehicle. 

For  such  a  field  test,  the  test  designer  must  select  the  optimum  number  of  observation 
opportunities  at  each  range.  The  designer  must  balance  collecting  enough  data  to  draw  valid 
conclusions  against  the  high  cost  of  supporting  vehicles  and  personnel  at  a  test  site. 
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2.  DESIGNING  THE  EXPERIMENT 


The  test  designer  selects  the  number  of  observations  in  order  to  achieve  two  objectives.  First,  the 
number  of  observations  must  be  large  enough  so  that  the  probabilities  of  errors  are  below  the 
desired  level.  And,  second,  the  number  of  observations  must  be  as  small  as  possible  to  minimize 
testing  time  and  cost. 

Table  I  illustrates  the  types  of  errors  that  can  occur  in  drawing  conclusions  from  an  observation 
test.  The  goal  of  an  observation  test  is  to  judge  whether  the  observed  difference  in  detection 
probability,  Pd ,  is  unusual  enough  to  reject  the  null  hypothesis  that  Vehicle  A  and  Vehicle  B 
have  the  same  Pd  .  If  the  analyst  concludes  that  the  null  hypothesis  cannot  be  rejected,  then 
either  the  analyst  has  made  a  correct  decision  or  a  Type  n  error.  On  the  other  hand,  a  decision  to 
reject  the  null  hypothesis  will  be  either  correct  or  a  Type  I  error.  In  terms  of  coimtermeasure 
effectiveness,  a  Typie  I  error  is  an  erroneous  conclusion  than  an  ineffective  countermeasure  is 
effective,  while  a  Type  II  error  miss  judges  an  effective  coimtermeasure. 

In  a  previous  paper  [1],  I  discussed  the  number  of  observation  opportunities  required  to  control 
the  probability  of  these  errors.  For  the  case  of  an  average  Pd  of  0.5,  Figure  2  plots  the  number  of 
observations  required  to  maintain  the  probability  of  Type  I  and  Type  II  errors  at  less  than  5%. 
For  example,  85  observations  of  each  vehicle  are  adequate  for  an  observed  0.15  difference  to  be 
significant  with  less  than  5%  chance  of  a  Type  I  error.  On  the  other  hand,  for  less  than  5% 
chance  of  committing  a  Type  11  error  when  the  underlying  probabilities  differ  by  0.15,  the 
requirement  is  316  observation  opportunities  per  vehicle. 

3.  AN  ITERATIVE  TECHNIQUE  FOR  TEST  DESIGN 

An  iterative  technique  can  reduce  the  number  of  observation  opportunities  required  for  a  test. 
Observation  data  is  described  by  the  binary  distribution.  But  the  binary  distribution  has  the 
property  that  the  standard  deviation  depends  upon  the  probability.  If  p  is  the  probability  and  N  is 
the  number  of  trials,  then  the  standard  deviation  of  the  number  of  successes,  o,  is  given  by 


a  =  yjNp(\-p)  (1) 

And  the  standard  deviation  of  the  proportion  of  successes  is  given  by 


IpC^-P)  (2) 

N  \  N 

Figure  3  plots  Equation  2  for  an  N  of  100, 200  and  300.  Figure  4  plots  the  same  equation 
normalized  to  its  maximum  at  p  =  0.5. 

Figure  5  further  shows  how  the  number  of  trials  change  with  probability  if  the  standard  deviation 
is  held  fixed.  For  exanple,  in  comparison  to  a  probability  of  0.5,  only  60%  as  many  trials  are 
need  at  a  probability  of  0. 1 . 


t 


To  use  this  property  of  the  binary  distribution,  the  test  designer  modifies  the  test  matrix  as  the 
test  progresses.  Early  test  results  yield  estimates  of  the  probability  of  detection  for  each  vehicle 
at  each  range.  Based  on  these  estimates,  the  test  designer  reallocates  the  number  of  observations 
among  the  ranges,  improving  the  efficiency  of  the  test  desigiL  Figure  5  illustrates  a  san^le  of  the 
improvement  in  efficiency  that  this  technique  can  achieve.  Initially  the  test  designer  would  have 
selected  the  number  of  observations  at  each  test  range  by  assuming  the  worst-case  value  of  Pd  = 
0.5.  But  as  the  test  progresses,  the  designer  would  use  estimates  of  Pd  from  the  early  measures  to 
redesign  the  test.  For  example,  at  range  20,  the  designer  would  reduce  the  number  of 
observations  from  314  to  251.  Overall  in  this  sample  test,  the  test  designer  would  reduce  the 
number  of  observations  from  1570  to  1 193,  a  reduction  of  24%. 

4.  IMPLEMENTATION  OF  THE  ITERATIVE  TECHNIQUE  FOR  TEST  DESIGN 

The  first  step  is  selection  of  the  number  of  observation  opportunities  for  each  range  based  on  the 
conservative  assumption  of  a  Pd  of  50%  at  each  range.  From  figures  x  and  y,  the  designer  selects 
No  to  reduce  the  probabilities  of  Type  I  and  II  errors  to  acceptable  levels. 

Next,  the  designer  calculates  estimates  of  Pj ,  the  average  probability  of  detection  at  range  Rj, 
based  on  the  results  of  the  first  day  of  observations,  as  follows: 


(3) 


where  Da  and  Db  are  the  numbers  of  detections  of  Vehicles  A  and  B  and  Na  and  Nb  are  the 
number  of  observation  opportunities  for  Vehicles  A  and  B. 

To  account  for  special  cases  when  there  are  no  detections  or  100%  detections.  Equation  3  should 
be  modified  to  prevent  later  setting  the  number  of  observation  opportunities  to  zero.  The 
following  form  covers  these  special  cases. 


If  (Da+Db)/(Na+Nb)  <  0.1,  then  Pj  =  0.1 
If  (Da+Db)/(Na+Nb)  >  0.9,  then  Pj  =  0.9 

Otherwise,  Pj  =  f/ 

(4) 


Next,  beginning  with  Equation  3,  the  designer  calculates  Ni,  the  new  number  of  observation 
opportunities  for  range  Ri: 


<7 


(4) 


Note  that  a  is  the  same  for  all  ranges.  Solving  for  Ni  yields. 


t 


N, = (5) 

CJ 

And,  keeping  the  total  number  of  observation  opportunities  constant  at  Nj ,  then 

(6) 

i=i  cr  ,=i 

Solving  for  a^,  gives 

ly-p  ,=i 

and 

'Epj(i-pj) 

7=1 

Now^,  the  test  continues  with  the  modified  number  of  opportunities  for  observation  at  each  range. 

At  the  end  of  each  succeeding  day,  the  designer  can  recalculate  the  Ni's  in  the  same  way  to  refine 
the  design. 


5.  CONCLUSION 

In  this  paper,  I  have  presented  an  iterative  technique  of  using  early  test  results  in  an  observation 
test  to  improve  the  overall  efficiency  of  the  test  design. 
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Possible  Outcomes  of  Hypothesis  Testing 


Is  Vehicle  A  Less  Detectable 
Than  Vehicle  B? 


Decision: 

No 

Yes 

Accept  Null  Hypothesis 

Correct 

Type  II  Error 

Accept  Alternative  Hypothesis 

Type  1  Error 

Correct 

Table  1.  Definitions  of  Type  I  and  Type  II  errors. 


Test  with  Fixed  Observers 


Fixed  Observers 


Figure  1.  Experimental  setup  for  an  observer  test. 


Number  of  Opportunities 
Required  to  Meet  Test  Criteria 
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Figure  2.  Number  of  observation  opportunities  required  to  meet  test  criteria. 
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Figure  3.  Standard  deviaticwi  of  a  binary  distribution  for  100, 200  and  300  trials. 
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Figure  4.  Standard  deviatim  of  a  binary  distribution  normalize  to  p-obability  of  0.5. 
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Figure  5.  An  example  of  iterative  design  of  an  observer  test. 


